test setup
Evaluating the Impact of Flaky Simulators on Testing Autonomous Driving Systems
Amini, Mohammad Hossein, Naseri, Shervin, Nejati, Shiva
Simulators are widely used to test Autonomous Driving Systems (ADS), but their potential flakiness can lead to inconsistent test results. We investigate test flakiness in simulation-based testing of ADS by addressing two key questions: (1) How do flaky ADS simulations impact automated testing that relies on randomized algorithms? and (2) Can machine learning (ML) effectively identify flaky ADS tests while decreasing the required number of test reruns? Our empirical results, obtained from two widely-used open-source ADS simulators and five diverse ADS test setups, show that test flakiness in ADS is a common occurrence and can significantly impact the test results obtained by randomized algorithms. Further, our ML classifiers effectively identify flaky ADS tests using only a single test run, achieving F1-scores of $85$%, $82$% and $96$% for three different ADS test setups. Our classifiers significantly outperform our non-ML baseline, which requires executing tests at least twice, by $31$%, $21$%, and $13$% in F1-score performance, respectively. We conclude with a discussion on the scope, implications and limitations of our study. We provide our complete replication package in a Github repository.
- North America > Canada > Ontario > National Capital Region > Ottawa (0.04)
- Oceania > Australia > Victoria > Melbourne (0.04)
- South America > Brazil (0.04)
- (6 more...)
- Transportation > Ground > Road (1.00)
- Information Technology > Robotics & Automation (1.00)
- Automobiles & Trucks (1.00)
Vectorized Scenario Description and Motion Prediction for Scenario-Based Testing
Winkelmann, Max, Vasconi, Constantin, Müller, Steffen
Automated vehicles (AVs) are tested in diverse scenarios, typically specified by parameters such as velocities, distances, or curve radii. To describe scenarios uniformly independent of such parameters, this paper proposes a vectorized scenario description defined by the road geometry and vehicles' trajectories. Data of this form are generated for three scenarios, merged, and used to train the motion prediction model VectorNet, allowing to predict an AV's trajectory for unseen scenarios. Predicting scenario evaluation metrics, VectorNet partially achieves lower errors than regression models that separately process the three scenarios' data. However, for comprehensive generalization, sufficient variance in the training data must be ensured. Thus, contrary to existing methods, our proposed method can merge diverse scenarios' data and exploit spatial and temporal nuances in the vectorized scenario description. As a result, data from specified test scenarios and real-world scenarios can be compared and combined for (predictive) analyses and scenario selection.
Autonomous Control for Orographic Soaring of Fixed-Wing UAVs
Suys, Tom, Hwang, Sunyou, de Croon, Guido C. H. E., Remes, Bart D. W.
Abstract-- We present a novel controller for fixed-wing UAVs that enables autonomous soaring in an orographic wind field, extending flight endurance. Our method identifies soaring regions and addresses position control challenges by introducing a target gradient line (TGL) on which the UAV achieves an equilibrium soaring position, where sink rate and updraft are balanced. We also demonstrate a single degree of control freedom in a soaring position through manipulation of the TGL. I. INTRODUCTION UAVs have benefited from advancements in battery technology and miniaturization of avionics, which resulted in an increase in their endurance and range. However, the full potential of UAV applications remains limited by reduced flight time.
- Transportation > Air (1.00)
- Energy (0.95)
- Aerospace & Defense > Aircraft (0.90)
Transfer Importance Sampling -- How Testing Automated Vehicles in Multiple Test Setups Helps With the Bias-Variance Tradeoff
Winkelmann, Max, Vasconi, Constantin, Müller, Steffen
The promise of increased road safety is a key motivator for the development of automated vehicles (AV). Yet, demonstrating that an AV is as safe as, or even safer than, a human-driven vehicle has proven to be challenging. Should an AV be examined purely virtually, allowing large numbers of fully controllable tests? Or should it be tested under real environmental conditions on a proving ground? Since different test setups have different strengths and weaknesses, it is still an open question how virtual and real tests should be combined. On the way to answer this question, this paper proposes transfer importance sampling (TIS), a risk estimation method linking different test setups. Fusing the concepts of transfer learning and importance sampling, TIS uses a scalable, cost-effective test setup to comprehensively explore an AV's behavior. The insights gained then allow parameterizing tests in a more trustworthy test setup accurately reflecting risks. We show that when using a trustworthy test setup alone is prohibitively expensive, linking it to a scalable test setup can increase efficiency $\unicode{x2013}$ without sacrificing the result's validity. Thus, the test setups' individual deficiencies are compensated for by their systematic linkage.
- Europe > Germany > Berlin (0.04)
- Oceania > New Zealand > North Island > Auckland Region > Auckland (0.04)
- North America > United States > Vermont > Washington County > Montpelier (0.04)
- (2 more...)
- Automobiles & Trucks (0.93)
- Transportation > Ground > Road (0.66)
Tunable Dynamic Walking via Soft Twisted Beam Vibration
Jiang, Yuhao, Chen, Fuchen, Aukes, Daniel M.
We propose a novel mechanism that propagates vibration through soft twisted beams, taking advantage of dynamically-coupled anisotropic stiffness to simplify the actuation of walking robots. Using dynamic simulation and experimental approaches, we show that the coupled stiffness of twisted beams with terrain contact can be controlled to generate a variety of complex trajectories by changing the frequency of the input signal. This work reveals how ground contact influences the system's dynamic behavior, supporting the design of walking robots inspired by this phenomenon. We also show that the proposed twisted beam produces a tunable walking gait from a single vibrational input.
- North America > United States > Arizona > Maricopa County > Mesa (0.04)
- North America > United States > Arizona > Maricopa County > Tempe (0.04)
Fast Calculation of the Knowledge Gradient for Optimization of Deterministic Engineering Simulations
van der Herten, Joachim, Couckuyt, Ivo, Deschrijver, Dirk, Dhaene, Tom
A novel efficient method for computing the Knowledge-Gradient policy for Continuous Parameters (KGCP) for deterministic optimization is derived. The differences with Expected Improvement (EI), a popular choice for Bayesian optimization of deterministic engineering simulations, are explored. Both policies and the Upper Confidence Bound (UCB) policy are compared on a number of benchmark functions including a problem from structural dynamics. It is empirically shown that KGCP has similar performance as the EI policy for many problems, but has better convergence properties for complex (multi-modal) optimization problems as it emphasizes more on exploration when the model is confident about the shape of optimal regions. In addition, the relationship between Maximum Likelihood Estimation (MLE) and slice sampling for estimation of the hyperparameters of the underlying models, and the complexity of the problem at hand, is studied.
- North America > United States > New York > New York County > New York City (0.04)
- Europe > Belgium > Flanders > East Flanders > Ghent (0.04)
- Information Technology > Artificial Intelligence > Representation & Reasoning > Optimization (1.00)
- Information Technology > Artificial Intelligence > Representation & Reasoning > Uncertainty > Bayesian Inference (0.86)
- Information Technology > Artificial Intelligence > Machine Learning > Learning Graphical Models > Directed Networks > Bayesian Learning (0.86)